Lessons Learned from Comparison Between Q-learning and Sarsa Agents in Bargaining Game

This paper focuses on sensitivity of learning mechanisms applied to agents in agent-based simulation and explores criteria for employing such learning mechanisms by comparing simulation results derived from agents who have different learning mechanisms. Specifically, we employ two types of reinforcement learning in this study, Q-learning and Sarsa. Through an analysis of simulation results in a bargaining game as one of the fundamental examples in game theory, the following implications have been revealed: (1) results between Q-learning and Sarsa agents are mostly the same from one viewpoint, but differ from another viewpoint; (2) Sarsa agents are superior to Q-learning agents in negotiation, while Sarsa agents cannot acquire rational behaviors, which can be acquired by Q-learning agents; and (3) implications (1) and (2) derive the following criteria that provide a rough guideline for employing learning mechanisms in agent-based simulation: (3-a) a discount factor in learning mechanisms and (3-b) comparison of simulation results mixing different learning agents. Contact: Keiki Takadama Department of Computational Intelligence and Systems Science Interdisciplinary Graduate School of Science and Engineering, Tokyo Institute of Technology 4259 Nagatsuta-cho, Midori-ku, Yokohama 226-8502 Japan Tel: +81-45-924-5204, Fax: +81-45-924-5219 Email: [email protected]